from sklearn_benchmarks.reporting.hp_match import HPMatchReporting

scikit-learn vs. ONNX Runtime

reporting = HPMatchReporting(against_lib="onnx", config="config.yml")
reporting.make_report()

We assume here there is a perfect match between the hyperparameters of both librairies. For a given set of parameters and a given dataset, we compute the speedup time scikit-learn / time onnx. For instance, a speedup of 2 means that onnx is twice as fast as scikit-learn for a given set of parameters and a given dataset.

KNeighborsClassifier_brute_force

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=brute.

WARNING! Mismatch between validation scores for 12 predictions (75.0%). See below.

	estimator	function	n_samples_train	n_samples	n_features	accuracy_score_sklearn	accuracy_score_onnx	diff_accuracy_scores
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	0.663	1.000	0.337
2	KNeighborsClassifier_brute_force	predict	100000	1	100	1.000	0.757	0.243
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	0.757	0.882	0.125
7	KNeighborsClassifier_brute_force	predict	100000	1000	100	0.882	1.000	0.118
8	KNeighborsClassifier_brute_force	predict	100000	1	100	1.000	0.757	0.243
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	0.882	0.663	0.219
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	0.757	1.000	0.243
14	KNeighborsClassifier_brute_force	predict	100000	1	100	1.000	0.922	0.078
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	0.663	0.929	0.266
19	KNeighborsClassifier_brute_force	predict	100000	1000	2	0.896	1.000	0.104
20	KNeighborsClassifier_brute_force	predict	100000	1	2	1.000	0.922	0.078
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	0.922	0.896	0.026

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.013	0.000	NaN	6.207	0.0	-1	1	NaN	19.345	0.136	0.663	0.001	0.001	See	See
3	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.013	0.000	NaN	6.176	0.0	-1	5	NaN	0.349	0.014	1.000	0.037	0.037	See	See
6	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.013	0.000	NaN	6.302	0.0	1	100	NaN	18.777	0.075	0.882	0.001	0.001	See	See
9	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.014	0.000	NaN	5.535	0.0	-1	100	NaN	0.351	0.011	1.000	0.041	0.041	See	See
12	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.013	0.001	NaN	6.126	0.0	1	5	NaN	4.263	0.048	0.896	0.003	0.003	See	See
15	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.013	0.000	NaN	5.974	0.0	1	1	NaN	0.296	0.012	1.000	0.045	0.045	See	See
18	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.005	0.000	NaN	0.334	0.0	-1	1	NaN	4.478	0.059	0.929	0.001	0.001	See	See
21	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.005	0.000	NaN	0.344	0.0	-1	5	NaN	0.294	0.008	1.000	0.016	0.016	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.059	0.120	NaN	0.000	0.002	-1	1	0.663	0.369	0.007	1.000	5.574	5.575	See	See
2	KNeighborsClassifier_brute_force	predict	100000	1	100	0.027	0.004	NaN	0.000	0.027	-1	1	1.000	18.934	0.194	0.757	0.001	0.001	See	See
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.870	0.062	NaN	0.000	0.003	-1	5	0.757	18.730	0.104	0.882	0.153	0.153	See	See
5	KNeighborsClassifier_brute_force	predict	100000	1	100	0.027	0.003	NaN	0.000	0.027	-1	5	1.000	0.361	0.007	1.000	0.075	0.075	See	See
7	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.235	0.026	NaN	0.000	0.002	1	100	0.882	0.355	0.014	1.000	6.291	6.296	See	See
8	KNeighborsClassifier_brute_force	predict	100000	1	100	0.025	0.001	NaN	0.000	0.025	1	100	1.000	18.699	0.044	0.757	0.001	0.001	See	See
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.878	0.070	NaN	0.000	0.003	-1	100	0.882	18.598	0.035	0.663	0.155	0.155	See	See
11	KNeighborsClassifier_brute_force	predict	100000	1	100	0.028	0.005	NaN	0.000	0.028	-1	100	1.000	0.361	0.005	1.000	0.078	0.078	See	See
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.242	0.020	NaN	0.000	0.002	1	5	0.757	0.282	0.008	1.000	7.965	7.968	See	See
14	KNeighborsClassifier_brute_force	predict	100000	1	100	0.025	0.002	NaN	0.000	0.025	1	5	1.000	4.377	0.043	0.922	0.006	0.006	See	See
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.416	0.018	NaN	0.001	0.001	1	1	0.663	4.630	0.044	0.929	0.306	0.306	See	See
17	KNeighborsClassifier_brute_force	predict	100000	1	100	0.022	0.001	NaN	0.000	0.022	1	1	1.000	0.289	0.018	1.000	0.078	0.078	See	See
19	KNeighborsClassifier_brute_force	predict	100000	1000	2	1.717	0.042	NaN	0.000	0.002	-1	1	0.896	0.292	0.013	1.000	5.879	5.885	See	See
20	KNeighborsClassifier_brute_force	predict	100000	1	2	0.004	0.000	NaN	0.000	0.004	-1	1	1.000	4.402	0.034	0.922	0.001	0.001	See	See
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	2.651	0.050	NaN	0.000	0.003	-1	5	0.922	4.375	0.052	0.896	0.606	0.606	See	See
23	KNeighborsClassifier_brute_force	predict	100000	1	2	0.008	0.003	NaN	0.000	0.008	-1	5	1.000	0.274	0.007	1.000	0.029	0.029	See	See

KNeighborsClassifier_kd_tree

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=kd_tree.

WARNING! Mismatch between validation scores for 12 predictions (75.0%). See below.

	estimator	function	n_samples_train	n_samples	n_features	accuracy_score_sklearn	accuracy_score_onnx	diff_accuracy_scores
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.929	1.000	0.071
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	1.000	0.946	0.054
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.946	0.951	0.005
7	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.951	1.000	0.049
8	KNeighborsClassifier_kd_tree	predict	1000000	1	10	1.000	0.946	0.054
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.951	0.929	0.022
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.946	1.000	0.054
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	1.000	0.911	0.089
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.929	0.894	0.035
19	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.891	1.000	0.109
20	KNeighborsClassifier_kd_tree	predict	1000	1	2	1.000	0.911	0.089
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.911	0.891	0.020

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	2.917	0.026	NaN	0.027	0.0	-1	1	NaN	132.394	0.000	0.929	0.022	0.022	See	See
3	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	4.631	0.071	NaN	0.017	0.0	-1	5	NaN	3.031	0.240	1.000	1.528	1.532	See	See
6	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	4.359	0.060	NaN	0.018	0.0	1	100	NaN	132.165	0.000	0.951	0.033	0.033	See	See
9	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	4.757	0.158	NaN	0.017	0.0	-1	100	NaN	3.160	0.275	1.000	1.506	1.511	See	See
12	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	4.356	0.145	NaN	0.018	0.0	1	5	NaN	0.044	0.001	0.891	98.715	98.742	See	See
15	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	4.622	0.054	NaN	0.017	0.0	1	1	NaN	0.006	0.000	1.000	744.782	746.905	See	See
18	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.001	NaN	0.018	0.0	-1	1	NaN	0.069	0.004	0.894	0.013	0.013	See	See
21	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.000	NaN	0.024	0.0	-1	5	NaN	0.006	0.000	1.000	0.116	0.116	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.851	1.033	NaN	0.0	0.001	-1	1	0.929	3.130	0.271	1.000	0.272	0.273	See	See
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.000	NaN	0.0	0.003	-1	1	1.000	131.694	0.000	0.946	0.000	0.000	See	See
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.094	0.351	NaN	0.0	0.001	-1	5	0.946	130.281	0.000	0.951	0.008	0.008	See	See
5	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	NaN	0.0	0.003	-1	5	1.000	3.122	0.247	1.000	0.001	0.001	See	See
7	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	5.830	0.494	NaN	0.0	0.006	1	100	0.951	3.022	0.207	1.000	1.930	1.934	See	See
8	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	NaN	0.0	0.003	1	100	1.000	129.656	0.000	0.946	0.000	0.000	See	See
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	3.376	0.283	NaN	0.0	0.003	-1	100	0.951	131.822	0.000	0.929	0.026	0.026	See	See
11	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.006	0.001	NaN	0.0	0.006	-1	100	1.000	3.148	0.283	1.000	0.002	0.002	See	See
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.880	0.392	NaN	0.0	0.002	1	5	0.946	0.006	0.000	1.000	327.859	328.311	See	See
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.002	0.000	NaN	0.0	0.002	1	5	1.000	0.047	0.001	0.911	0.039	0.039	See	See
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.031	0.288	NaN	0.0	0.001	1	1	0.929	0.068	0.001	0.894	15.068	15.071	See	See
17	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.001	0.000	NaN	0.0	0.001	1	1	1.000	0.006	0.000	1.000	0.192	0.192	See	See
19	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.035	0.016	NaN	0.0	0.000	-1	1	0.891	0.006	0.000	1.000	5.931	5.940	See	See
20	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.002	0.000	NaN	0.0	0.002	-1	1	1.000	0.045	0.002	0.911	0.055	0.055	See	See
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.034	0.003	NaN	0.0	0.000	-1	5	0.911	0.047	0.003	0.891	0.724	0.726	See	See
23	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.003	0.001	NaN	0.0	0.003	-1	5	1.000	0.006	0.000	1.000	0.498	0.499	See	See

HistGradientBoostingClassifier_best

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: learning_rate=0.01, n_iter_no_change=10.0, max_leaf_nodes=100.0, max_bins=255.0, min_samples_leaf=100.0, max_iter=300.0.

WARNING! Mismatch between validation scores for 1 prediction (100.0%). See below.

	estimator	function	n_samples_train	n_samples	n_features	accuracy_score_sklearn	accuracy_score_onnx	diff_accuracy_scores
1	HistGradientBoostingClassifier_best	predict	100000	1000	100	0.824	1.0	0.176

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	HistGradientBoostingClassifier_best	fit	100000	100000	100	129.223	0.0	300	0.001	0.001	NaN	0.556	0.032	0.824	232.305	232.678	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	HistGradientBoostingClassifier_best	predict	100000	1000	100	0.157	0.003	300	0.005	0.0	0.824	0.466	0.018	1.0	0.337	0.337	See	See